[GRPC] Bulk optimization and fixes by karenyrx · Pull Request #19937 · opensearch-project/OpenSearch

karenyrx · 2025-11-09T05:38:52Z

Description

A series of gRPC bulk API fixes/optimizations:

Fix update operation to use doc field instead of object field (still fallsback to object for backward compatibility)
Fix the default value of fetchSource to null to match REST
Add pipeline support for upsert
Optimize document bytes copying to pass bytes reference

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

Functionality includes testing.
API changes companion pull request created, if applicable.
Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

…llsback to object for bwc) bytes optimize Set the default value of source to null to match REST Support allowExplicitIndex setting Signed-off-by: Karen Xu <karenxyr@gmail.com>

github-actions · 2025-11-09T05:51:33Z

❌ Gradle check result for f3818ca: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Karen Xu <karenxyr@gmail.com>

github-actions · 2025-11-10T04:35:08Z

❕ Gradle check result for 34bbaf4: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

codecov · 2025-11-10T04:35:28Z

Codecov Report

❌ Patch coverage is 91.83673% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 73.24%. Comparing base (2f930dc) to head (2fa7426).
⚠️ Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
...est/document/bulk/BulkRequestParserProtoUtils.java	91.66%	0 Missing and 4 partials ⚠️

Additional details and impacted files

@@            Coverage Diff            @@
##               main   #19937   +/-   ##
=========================================
  Coverage     73.24%   73.24%           
- Complexity    71519    71561   +42     
=========================================
  Files          5789     5789           
  Lines        327121   327142   +21     
  Branches      47117    47120    +3     
=========================================
+ Hits         239591   239608   +17     
- Misses        68269    68324   +55     
+ Partials      19261    19210   -51

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Signed-off-by: Karen Xu <karenxyr@gmail.com>

github-actions · 2025-11-10T23:18:47Z

❌ Gradle check result for 6d897b8: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-11-11T06:02:43Z

❌ Gradle check result for 6d897b8: null

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

...a/org/opensearch/transport/grpc/proto/request/document/bulk/BulkRequestParserProtoUtils.java

Signed-off-by: Karen Xu <karenxyr@gmail.com>

github-actions · 2025-11-12T17:52:02Z

❌ Gradle check result for 2a16f30: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

...a/org/opensearch/transport/grpc/proto/request/document/bulk/BulkRequestParserProtoUtils.java

github-actions · 2025-11-12T20:36:45Z

❕ Gradle check result for 2a16f30: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

Signed-off-by: Karen Xu <karenxyr@gmail.com>

github-actions · 2025-11-13T01:16:22Z

✅ Gradle check result for 3b80f63: SUCCESS

...a/org/opensearch/transport/grpc/proto/request/document/bulk/BulkRequestParserProtoUtils.java

msfroh

Looks good to me overall. I really appreciate the use of zero-copy for the byte arrays.

I'll hold off on merging until you've had a chance to respond to @varunbharadwaj's comments.

...a/org/opensearch/transport/grpc/proto/request/document/bulk/BulkRequestParserProtoUtils.java

github-actions · 2025-11-19T00:37:35Z

❌ Gradle check result for 9e0aae9: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Karen X <karenxyr@gmail.com>

github-actions · 2025-11-19T01:52:43Z

❌ Gradle check result for 2fa7426: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-11-19T04:41:27Z

✅ Gradle check result for 2fa7426: SUCCESS

* Add pipeline, fix update to use doc instead of object field (still fallsback to object for bwc) bytes optimize Set the default value of source to null to match REST Support allowExplicitIndex setting Signed-off-by: Karen Xu <karenxyr@gmail.com> * fix tests Signed-off-by: Karen Xu <karenxyr@gmail.com> * changelog Signed-off-by: Karen Xu <karenxyr@gmail.com> * add tests Signed-off-by: Karen Xu <karenxyr@gmail.com> * remove allowExplicitIndex Signed-off-by: Karen Xu <karenxyr@gmail.com> * add bytesref to index/create ops too Signed-off-by: Karen Xu <karenxyr@gmail.com> * toBytesArray Signed-off-by: Karen X <karenxyr@gmail.com> --------- Signed-off-by: Karen Xu <karenxyr@gmail.com> Signed-off-by: Karen X <karenxyr@gmail.com>

Add pipeline, fix update to use doc instead of object field (still fa…

d920cde

…llsback to object for bwc) bytes optimize Set the default value of source to null to match REST Support allowExplicitIndex setting Signed-off-by: Karen Xu <karenxyr@gmail.com>

fix tests

113d3ed

Signed-off-by: Karen Xu <karenxyr@gmail.com>

karenyrx force-pushed the bulkfixes branch from f3818ca to 113d3ed Compare November 10, 2025 03:16

changelog

34bbaf4

Signed-off-by: Karen Xu <karenxyr@gmail.com>

opensearch-ci-bot mentioned this pull request Nov 10, 2025

[AUTOCUT] Gradle Check Flaky Test Report for ClientYamlTestSuiteIT #14319

Open

karenyrx marked this pull request as ready for review November 10, 2025 21:50

karenyrx requested a review from a team as a code owner November 10, 2025 21:50

add tests

6d897b8

Signed-off-by: Karen Xu <karenxyr@gmail.com>

karenyrx force-pushed the bulkfixes branch from b466a94 to 6d897b8 Compare November 10, 2025 22:11

opensearch-ci-bot mentioned this pull request Nov 11, 2025

[AUTOCUT] Gradle Check Flaky Test Report for FullClusterRestartIT #15813

Open

finnegancarroll reviewed Nov 11, 2025

View reviewed changes

...a/org/opensearch/transport/grpc/proto/request/document/bulk/BulkRequestParserProtoUtils.java Outdated Show resolved Hide resolved

...a/org/opensearch/transport/grpc/proto/request/document/bulk/BulkRequestParserProtoUtils.java Show resolved Hide resolved

remove allowExplicitIndex

2a16f30

Signed-off-by: Karen Xu <karenxyr@gmail.com>

opensearch-ci-bot mentioned this pull request Nov 12, 2025

[AUTOCUT] Gradle Check Flaky Test Report for QueryPhaseTests #14551

Open

finnegancarroll approved these changes Nov 12, 2025

View reviewed changes

varunbharadwaj reviewed Nov 12, 2025

View reviewed changes

...a/org/opensearch/transport/grpc/proto/request/document/bulk/BulkRequestParserProtoUtils.java Show resolved Hide resolved

opensearch-ci-bot mentioned this pull request Nov 12, 2025

[AUTOCUT] Gradle Check Flaky Test Report for AwarenessAllocationIT #17930

Open

add bytesref to index/create ops too

3b80f63

Signed-off-by: Karen Xu <karenxyr@gmail.com>

varunbharadwaj reviewed Nov 13, 2025

View reviewed changes

...a/org/opensearch/transport/grpc/proto/request/document/bulk/BulkRequestParserProtoUtils.java Show resolved Hide resolved

varunbharadwaj reviewed Nov 13, 2025

View reviewed changes

...a/org/opensearch/transport/grpc/proto/request/document/bulk/BulkRequestParserProtoUtils.java Outdated Show resolved Hide resolved

varunbharadwaj reviewed Nov 13, 2025

View reviewed changes

...a/org/opensearch/transport/grpc/proto/request/document/bulk/BulkRequestParserProtoUtils.java Outdated Show resolved Hide resolved

msfroh approved these changes Nov 13, 2025

View reviewed changes

...a/org/opensearch/transport/grpc/proto/request/document/bulk/BulkRequestParserProtoUtils.java Show resolved Hide resolved

karenyrx force-pushed the bulkfixes branch from 7f3ec5a to 984bdd3 Compare November 19, 2025 00:18

karenyrx force-pushed the bulkfixes branch from 984bdd3 to fffdcca Compare November 19, 2025 00:29

toBytesArray

57bc645

Signed-off-by: Karen X <karenxyr@gmail.com>

karenyrx force-pushed the bulkfixes branch from 9e0aae9 to 57bc645 Compare November 19, 2025 00:40

Merge branch 'main' into bulkfixes

2fa7426

Signed-off-by: Karen X <karenxyr@gmail.com>

varunbharadwaj approved these changes Nov 19, 2025

View reviewed changes

opensearch-ci-bot mentioned this pull request Nov 19, 2025

[AUTOCUT] Gradle Check Flaky Test Report for PruneFileCacheIT #19724

Closed

karenyrx merged commit 6a54b0b into opensearch-project:main Nov 19, 2025
32 of 34 checks passed

This was referenced Nov 19, 2025

[AUTOCUT] Gradle Check Flaky Test Report for RemoteRestoreSnapshotIT #14324

Open

[AUTOCUT] Gradle Check Flaky Test Report for ClusterDisruptionIT #14308

Open

BrewTestBot mentioned this pull request Dec 17, 2025

opensearch 3.4.0 Homebrew/homebrew-core#258986

Merged

tkykenmt mentioned this pull request Jan 11, 2026

[feature] GRPC Transport Bugfixes tkykenmt/opensearch-feature-explorer#1718

Closed

3 tasks

Conversation

karenyrx commented Nov 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issues

Check List

Uh oh!

github-actions bot commented Nov 9, 2025

Uh oh!

github-actions bot commented Nov 10, 2025

Uh oh!

codecov bot commented Nov 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions bot commented Nov 10, 2025

Uh oh!

github-actions bot commented Nov 11, 2025

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Nov 12, 2025

Uh oh!

Uh oh!

github-actions bot commented Nov 12, 2025

Uh oh!

github-actions bot commented Nov 13, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

msfroh left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions bot commented Nov 19, 2025

Uh oh!

github-actions bot commented Nov 19, 2025

Uh oh!

github-actions bot commented Nov 19, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Comments

karenyrx commented Nov 9, 2025 •

edited

Loading

codecov bot commented Nov 10, 2025 •

edited

Loading